Linked advertisements

ABSTRACT

Embodiments of the present invention generate linked advertisements. Linked advertisements may include a preliminary advertisement and one or more subsequent advertisements. In one embodiment, the viewer is only shown the subsequent advertisement upon detecting a positive reaction to the preliminary advertisement. The subsequent advertisement is associated with presentation triggers that specify a context in which the subsequent advertisement should be presented. The context may be defined by a time of day, location, user activity, and/or other parameters. For example, the presentation trigger may specify a context in which the user is able to purchase an advertised good or service. For example, the context could be the user driving (activity context) near a coffee shop (location context) in the morning (time context). The reaction to the preliminary advertisement may be explicit or implicit.

BACKGROUND

Advertisements are shown before, during, and after media presentations.Advertisements are even included within media presentations throughproduct placement. The advertisements shown with the media are selectedbased on anticipated audience demographics and interests of theanticipated audience. The advertisements are shown regardless of whetherit is a good time for the audience member to act in response to theadvertisement.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Embodiments of the present invention generate linked advertisements.Linked advertisements may include a preliminary advertisement and one ormore subsequent advertisements. In one embodiment, the viewer is onlyshown the subsequent advertisement upon detecting a positive reaction tothe preliminary advertisement. The subsequent advertisement isassociated with presentation triggers that specify a context in whichthe subsequent advertisement should be presented. The context may bedefined by a time of day, location, user activity, and/or otherparameters. For example, the presentation trigger may specify a contextin which the user is able to purchase an advertised good or service. Forexample, the context could be the user driving (activity context) near acoffee shop (location context) in the morning (time context).

The preliminary advertisement and subsequent advertisement may be shownon different devices. For example, the preliminary advertisement may beshown on a television as part of a media presentation (e.g., a separatead, product placement) and the subsequent advertisement could be shownon a mobile device, such as a smartphone or a tablet. Showing thesubsequent advertisement on a location-aware (e.g., GPS enabled) mobiledevice allows the presentation trigger to include a location.

The reaction to the preliminary advertisement may be explicit orimplicit. An explicit reaction could be the user making an affirmativegesture indicating that he likes the preliminary advertisement. Theimplicit reaction may be derived through an analysis of image data. Theimage data may be generated by a depth camera, video camera, or otherimaging device. In one embodiment, the user's facial expressions areanalyzed to determine a reaction to an advertised product or service.Apart from, or in combination with the facial expressions, biometricreadings may be derived from the imaging data.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in detail below withreference to the attached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor implementing embodiments of the invention;

FIG. 2 is a diagram of an online entertainment environment, inaccordance with an embodiment of the present invention;

FIG. 3 is a diagram of a remote entertainment computing environment, inaccordance with an embodiment of the present invention;

FIG. 4 is a diagram of an exemplary audience area captured using a depthcamera, in accordance with an embodiment of the present invention;

FIG. 5 is a diagram of an exemplary audience area captured using a depthcamera, in accordance with an embodiment of the present invention;

FIG. 6 is a diagram of an exemplary audience area captured using a depthcamera, in accordance with an embodiment of the present invention;

FIG. 7 is a diagram showing an ad path, in accordance with an embodimentof the present invention;

FIG. 8 is a diagram showing an ad path, in accordance with an embodimentof the present invention;

FIG. 9 is a flow chart showing a method of providing linkedadvertisements, in accordance with an embodiment of the presentinvention;

FIG. 10 is a flow chart showing a method of assigning an audience memberto an ad path, in accordance with an embodiment of the presentinvention; and

FIG. 11 is a flow chart showing a method of managing an ad path, inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The subject matter of embodiments of the invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Embodiments of the present invention generate linked advertisements.Linked advertisements may include a preliminary advertisement and one ormore subsequent advertisements. The preliminary advertisement andsubsequent advertisement may be separated by time, location, and device.The preliminary advertisement and subsequent advertisements displayrelated products or services and may operate together as a unified adcampaign.

In one embodiment, the viewer is only shown the subsequent advertisementupon detecting a positive reaction to the preliminary advertisement. Thesubsequent advertisement is associated with presentation triggers thatspecify a context in which the subsequent advertisement should bepresented. The context may be defined by a time of day, location, useractivity, and/or other parameters. For example, the presentation triggermay specify a context in which the user is able to purchase anadvertised good or service. For example, the context could be the userdriving (activity context) near a coffee shop (location context) in themorning (time context).

The preliminary advertisement and subsequent advertisement may be shownon different devices. For example, the preliminary advertisement may beshown on a television as part of a media presentation (e.g., a separatead, product placement) and the subsequent advertisement could be shownon a mobile device, such as a smartphone or a tablet. Showing thesubsequent advertisement on a location-aware (e.g., GPS enabled) mobiledevice allows the presentation trigger to include a location. Othercontextual parameters may include a time of day and the user's currentactivity. For example, the presentation trigger could specify that thesubsequent advertisement is only shown during business hours for theretail outlet. In another example, the subsequent advertisement is onlyshown at a time when the user is likely to purchase a product orservice. For example, a user may be likely to purchase food at arestaurant during lunch time or dinner time.

The reaction to the preliminary advertisement may be explicit orimplicit. An explicit reaction could be the user making an affirmativegesture indicating that they like the preliminary advertisement. Theuser could also explicitly request more information or otherwise expressan interest in the advertised product through a companion device, suchas a smartphone or tablet. In one embodiment, a companion application isprovided to allow the user to explicitly indicate he likes an advertisedproduct or service. The subsequent advertisement may include a coupon orother incentive for the user to try the product or service. In this way,the user could be encouraged to express her interest in an advertisedproduct or service through the companion application. For users that donot have a companion device, or do not want to use a companionapplication, an explicit indication of interest may be made through agesture, game pad, keyboard, controller, or voice command picked up byan entertainment device facilitating the linked advertising. Theentertainment device could be a television, game console, cable box, orother similar device that is able to receive input from the audience andcorrelate it with what content the display device shows.

In one embodiment, the user's reaction to an advertised product (e.g., aseparate ad, product placement) or service is implicit. The implicitreaction may be derived through an analysis of image data. A depthcamera, video camera, or other imaging device may generate the imagedata. In one embodiment, the user's facial expressions are analyzed todetermine a reaction to an advertised product or service. Apart from, orin combination with, the facial expressions, biometric readings may bederived from the imaging data. For example, facial flushing and heartrate may be determined from the imaging data and used to classify thereaction as positive, negative, or indifferent. In one embodiment, theaudience member's facial expressions and biometric changes are comparedagainst a baseline for the audience member to determine whether thereaction is positive as well as a strength of the reaction.

The subsequent advertisements may be part of an advertising path thatincludes a series of advertisements with different presentationtriggers, content, and incentives. In one embodiment, the strength ofthe user's reaction to the preliminary advertisement is used to activatedifferent advertisements within the path. The presentation trigger foran active subsequent advertisement is monitored, whereas presentationtriggers for inactive advertisements are not. Ads within the path may beactivated and deactivated in response to additional user actions orrules.

In one embodiment, a strong positive reaction to the preliminary adactivates a subsequent advertisement having a comparatively lowerincentive. For example, the subsequent advertisement may include a 50cent discount on a sandwich. A mild or weekly positive reaction to thepreliminary advertisement may activate a subsequent advertisement havinga higher incentive. For example, the subsequent advertisement could havea two dollar discount on a sandwich.

The user's prior purchase history may also be used to determine whichad(s) in an ad path to active. For example, if a user repeatedly ignoresan advertisement with a lower incentive, he may be moved to anadvertisement with a higher incentive. Similarly, if a user gives astrong positive response to a preliminary advertisement, but is known toregularly purchase products associated with the advertisement, he may beassociated with a subsequent advertisement having a lower incentive.Alternatively, the user could be associated with a subsequentadvertisement that reminds the user of a consumer club he is in, such asa sandwich club. This subsequent advertisement could remind the userthat he needs to purchase two more sandwiches before he earns a freesandwich.

Embodiments of the present invention use audience data to select anappropriate ad path from one of several ad paths available. The audiencedata may be derived from image data generated by an imaging device, suchas a video camera, that has a view of the audience area. Automated imageanalysis may be used to generate useful audience data that is used toselect the overlay.

The audience data derived from the image data includes number of peoplepresent in the audience, engagement level of people in the audience,personal characteristics of those individuals, and response to the mediacontent. Different levels of engagement may be assigned to audiencemembers. Image data may be analyzed to determine how many people arepresent in the audience and characteristics of those people.

Audience data includes a level of engagement or attentiveness. Aperson's attentiveness may be classified into one or more categories orlevels. The categories may range from not paying attention to fullattention. A person that is not looking at the television and is in aconversation with somebody else, either in the room or on the phone, maybe classified as not paying attention or fully distracted. On the otherhand, somebody in the room that is not looking at the TV, but is nototherwise obviously distracted, may have a medium level ofattentiveness. Someone that is looking directly at the televisionwithout an apparent distraction may be classified as fully attentive. Amachine-learning image classifier may assign the levels of attentivenessby analyzing image data.

Audience data may include a person's reaction to a media content, suchas a preliminary advertisement. The person's reaction may be measured bystudying biometrics gleaned from the imaging data. For example,heartbeat and facial flushing may be detected in the image data.Similarly, pupil dilation and other facial expressions may be associatedwith different reactions. All of these biometric characteristics may beinterpreted by a classifier to determine whether the person likes ordislikes a media content.

The different audience data may be used to determine when reactioncriteria associated with a preliminary advertisement are satisfied. Forexample, a criterion may not be satisfied when a person is present butshows a low level of attentiveness. An advertiser may specify that an adpath is activated only when one or more of the individuals present arefully attentive.

A person's reaction to a primary ad or other media content may be usedto determine whether a subsequent ad is activated. For example, a personclassified as having a negative reaction to a product placement within amovie may not be associated with an ad path for the product advertisedthrough product placement. Alternatively, a person that respondspositively to a primary ad may be associated with an ad path for arelated product or service.

In addition to determining whether to activate an ad path based onengagement levels, personal characteristics of audience members may alsobe considered when selecting an ad path. The personal characteristics ofthe audience members include demographic data that may be discerned fromimage classification or from associating the person with a knownpersonal account. For example, an entertainment company may require thatthe person submit a name, age, address, and other demographicinformation to maintain a personal account. The personal account may beassociated with a facial recognition program that is used toauthenticate the person. Regardless of whether the entertainment companyis providing the primary ad, the facial recognition record associatedwith the personal account could be used to identify the person in theaudience associated with the account. In some situations, all of theaudience members may be associated with an account that allows precisedemographic information to be associated with each audience member.Account information may be used to associate multiple devices with anaudience member.

Having briefly described an overview of embodiments of the invention, anexemplary operating environment suitable for use in implementingembodiments of the invention is described below.

Exemplary Operating Environment

Referring to the drawings in general, and initially to FIG. 1 inparticular, an exemplary operating environment for implementingembodiments of the invention is shown and designated generally ascomputing device 100. Computing device 100 is but one example of asuitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing device 100 be interpreted as having anydependency or requirement relating to any one or combination ofcomponents illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program components, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program components, including routines, programs,objects, components, data structures, and the like, refer to code thatperforms particular tasks or implements particular abstract data types.Embodiments of the invention may be practiced in a variety of systemconfigurations, including handheld devices, consumer electronics,general-purpose computers, specialty computing devices, etc. Embodimentsof the invention may also be practiced in distributed computingenvironments where tasks are performed by remote-processing devices thatare linked through a communications network.

With continued reference to FIG. 1, computing device 100 includes a bus110 that directly or indirectly couples the following devices: memory112, one or more processors 114, one or more presentation components116, input/output (I/O) ports 118, I/O components 120, and anillustrative power supply 122. Bus 110 represents what may be one ormore busses (such as an address bus, data bus, or combination thereof).Although the various blocks of FIG. 1 are shown with lines for the sakeof clarity, in reality, delineating various components is not so clear,and metaphorically, the lines would more accurately be grey and fuzzy.For example, one may consider a presentation component such as a displaydevice to be an I/O component 120. Also, processors have memory. Theinventors hereof recognize that such is the nature of the art, andreiterate that the diagram of FIG. 1 is merely illustrative of anexemplary computing device that can be used in connection with one ormore embodiments of the invention. Distinction is not made between suchcategories as “workstation,” “server,” “laptop,” “handheld device,”etc., as all are contemplated within the scope of FIG. 1 and refer to“computer” or “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data.

Computer storage media includes RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices. Computer storage media doesnot comprise a propagated data signal.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory 112 may be removable,nonremovable, or a combination thereof. Exemplary memory includessolid-state memory, hard drives, optical-disc drives, etc. Computingdevice 100 includes one or more processors 114 that read data fromvarious entities such as bus 110, memory 112 or I/O components 120.Presentation component(s) 116 present data indications to a person orother device. Exemplary presentation components 116 include a displaydevice, speaker, printing component, vibrating component, etc. I/O ports118 allow computing device 100 to be logically coupled to other devicesincluding I/O components 120, some of which may be built in.Illustrative I/O components 120 include a microphone, joystick, gamepad, satellite dish, scanner, printer, wireless device, etc.

Exemplary Entertainment Environment

Turning now to FIG. 2, an online entertainment environment 200 is shown,in accordance with an embodiment of the present invention. The onlineentertainment environment 200 comprises various entertainment devicesconnected through a network 220 to an entertainment service 230.Exemplary entertainment devices include a game console 210, a tablet212, a personal computer 214, a digital video recorder 217, a cable box218, and a television 216. Use of other entertainment devices notdepicted in FIG. 2, such as smart phones, is also possible.

The game console 210 may have one or more game controllerscommunicatively coupled to it. In one embodiment, the tablet 212 may actas an input device for the game console 210 or the personal computer214. In another embodiment, the tablet 212 is a stand-aloneentertainment device. Network 220 may be a wide area network, such asthe Internet. As can be seen, most devices shown in FIG. 2 could bedirectly connected to the network 220. The devices shown in FIG. 2, areable to communicate with each other through the network 220 and/ordirectly as indicated by the lines connecting the devices.

The controllers associated with game console 210 include a game pad 211,a headset 236, an imaging device 213, and a tablet 212. Tablet 212 isshown coupled directly to the game console 210, but the connection couldbe indirect through the Internet or a subnet. In one embodiment, theentertainment service 230 helps make a connection between the tablet 212and the game console 210. The tablet 212 is capable of generatingnumerous input streams and may also serve as a display output mechanism.In addition to being a primary display, the tablet 212 could providesupplemental information related to primary information shown on aprimary display, such as television 216. The input streams generated bythe tablet 212 include video and picture data, audio data, movementdata, touch screen data, and keyboard input data.

The headset 236 captures audio input from a player and the player'ssurroundings and may also act as an output device, if it is coupled witha headphone or other speaker. The headset 236 may facilitate voicecontrol of the game console or other entertainment devices. A microphone(not shown) may be integrated into or connected to any of theentertainment devices to facilitate voice control.

The imaging device 213 is coupled to game console 210. The imagingdevice 213 may be a video camera, a still camera, a depth camera, or avideo camera capable of taking still or streaming images. In oneembodiment, the imaging device 213 includes an infrared light and aninfrared camera. The imaging device 213 may also include a microphone,speaker, and other sensors. In one embodiment, the imaging device 213 isa depth camera that generates three-dimensional image data. Thethree-dimensional image data may be a point cloud or depth cloud. Thethree-dimensional image data may associate individual pixels with bothdepth data and color data. For example, a pixel within the depth cloudmay include red, green, and blue color data, and X, Y, and Zcoordinates. Stereoscopic depth cameras are also possible. The imagingdevice 213 may have several image-gathering components. For example, theimaging device 213 may have multiple cameras. In other embodiments, theimaging device 213 may have multidirectional functionality. In this way,the imaging device 213 may be able to expand or narrow a viewing rangeor shift its viewing range from side to side and up and down.

The game console 210 may have image-processing functionality that iscapable of identifying objects within the depth cloud. For example,individual people may be identified along with characteristics of theindividual people. In one embodiment, gestures made by the individualpeople may be distinguished and used to control games or media output bythe game console 210. The game console 210 may use the image data,including depth cloud data, for facial recognition purposes tospecifically identify individuals within an audience area. The facialrecognition function may associate individuals with an accountassociated with a gaming service or media service, or used for loginsecurity purposes, to specifically identify the individual.

In one embodiment, the game console 210 uses microphone, and/or imagedata captured through imaging device 213 to identify content beingdisplayed through television 216. For example, a microphone may pick upthe audio data of a movie being generated by the cable box 218 anddisplayed on television 216. The audio data may be compared with adatabase of known audio data and the data identified using automaticcontent recognition techniques, for example. Content being displayedthrough the tablet 212 or the PC 214 may be identified in a similarmanner. In this way, the game console 210 is able to determine what ispresently being displayed to a person regardless of whether the gameconsole 210 is the device generating and/or distributing the content fordisplay.

The game console 210 may include classification programs that analyzeimage and/or data to generate audience data. For example, the gameconsole 210 may determine number of people in the audience, audiencemember characteristics, levels of engagement, and audience response.Audio data may be compared with a content data base do identify contentbeing displayed. The audio data may be captured by a microphone coupledto the game console 210. In this way, the content displayed may beidentified when it is output by an entertainment device other than thegame console 210. Content output by the game console 210 could also beidentified using the audio signals.

In another embodiment, the game console 210 includes a local storagecomponent. The local storage component may store user profiles forindividual persons or groups of persons viewing and/or reacting to mediacontent. Each user profile may be stored as a separate file, such as acookie. The information stored in the user profiles may be updatedautomatically. Personal information, viewing histories, viewingselections, personal preferences, the number of times a person hasviewed known media content, the portions of known media content theperson has viewed, a person's responses to known media content, and aperson's engagement levels in known media content may be stored in auser profile associated with a person. As described elsewhere, theperson may be first identified before information is stored in a userprofile associated with the person. In other embodiments, a person'scharacteristics may be first recognized and mapped to an existing userprofile for a person with similar or the same characteristics.Demographic information may also be stored. Each item of information maybe stored as a “viewing record” associated with a particular type ofmedia content. As well, viewer personas, as described below, may bestored in a user profile.

Entertainment service 230 may comprise multiple computing devicescommunicatively coupled to each other. In one embodiment, theentertainment service is implemented using one or more server farms. Theserver farms may be spread out across various geographic regionsincluding cities throughout the world. In this scenario, theentertainment devices may connect to the closest server farms.Embodiments of the present invention are not limited to this setup. Theentertainment service 230 may provide primary content and secondarycontent. Primary content may include television shows, movies, and videogames. Secondary content may include advertisements, social content,directors' information and the like.

FIG. 2 also includes a cable box 218 and a DVR 217. Both of thesedevices are capable of receiving content through network 220. Thecontent may be on-demand or broadcast as through a cable distributionnetwork. Both the cable box 218 and DVR 217 have a direct connectionwith television 216. Both devices are capable of outputting content tothe television 216 without passing through game console 210, but in oneembodiment the cable box 218 and DVR 217 pass through the game console210. As can be seen, game console 210 also has a direct connection totelevision 216. Television 216 may be a smart television that is capableof receiving entertainment content directly from entertainment service230. As mentioned, the game console 210 may perform audio analysis todetermine what media title is being output by the television 216 whenthe title originates with the cable box 218, DVR 217, or television 216.

Exemplary Advertising and Content Service

Turning now to FIG. 3, a distributed entertainment environment 300 isshown, in accordance with an embodiment of the present invention. Theentertainment environment 300 includes entertainment device A 310,entertainment device B 312, entertainment device C 314, andentertainment device N 316 (hereafter entertainment devices 310-316).Entertainment device N 316 is intended to represent that there could bean almost unlimited number of clients connected to network 305. Theentertainment devices 310-316 may take different forms. For example, theentertainment devices 310-316 may be game consoles, televisions, DVRs,cable boxes, personal computers, tablets, or other entertainment devicescapable of outputting media. In addition, the entertainment devices310-316 are capable of gathering viewer data through an imaging device,similar to imaging device 213 of FIG. 2 that was previously described.The imaging device could be built into a client, such as a web cam andmicrophone, or could be a stand-alone device.

In one embodiment, the entertainment devices 310-316 include a localstorage component configured to store personal profiles for one or morepersons. The local storage component is described in greater detailabove with reference to the game console 210. The entertainment devices310-316 may include classification programs that analyze image data togenerate audience data. For example, the entertainment devices 310-316may determine how many people are in the audience, audience membercharacteristics, levels of engagement, and audience response.

Network 305 is a wide area network, such as the Internet. Network 305 isconnected to advertiser 320, content provider 322, and secondary contentprovider 324. The advertiser 320 distributes advertisements toentertainment devices 310-316. The advertiser 320 may also cooperatewith entertainment service 330 to provide advertisements. The contentprovider 322 provides primary content such as movies, video games, andtelevision shows. The primary content may be provided directly toentertainment devices 310-316 or indirectly through entertainmentservice 330.

Secondary content provider 324 provides content that compliments theprimary content. Secondary content may be a director's cut, informationabout a character, game help information, and other content thatcompliments the primary content. The same entity may generate bothprimary content and secondary content. For example, a television showmay be generated by a director that also generates additional secondarycontent to compliment the television show. The secondary content andprimary content may be purchased separately and could be displayed ondifferent devices. For example, the primary content could be displayedthrough a television while the secondary content is viewed on acompanion device, such as a tablet. The advertiser 320, content provider322, and secondary content provider 324 may stream content directly toentertainment devices or seek to have their content distributed by aservice, such as entertainment service 330.

Entertainment service 330 provides content and advertisements toentertainment devices. The entertainment service 330 is shown as asingle block. In reality, the functions should be widely distributedacross multiple devices. In embodiments of the present invention, thevarious features of entertainment service 330 described herein may beprovided by multiple entities and components. The entertainment service330 comprises a game execution environment 332, a game data store 334, acontent data store 336, a distribution component 338, a streamingcomponent 340, a content recognition database 342, an ad data store 344,an ad placement component 346, an ad sales component 348, an audiencedata store 350, an audience processing component 352, and an audiencedistribution component 354. As can be seen, the various components maywork together to provide content, including games, advertisements, andmedia titles to a client, and capture audience data. The audience datamay be used to specifically target advertisements and/or content to aperson. The audience data may also be aggregated and shared with or soldto others.

The game execution environment 332 provides an online gaming experienceto a client device. The game execution environment 332 comprises thegaming resources required to execute a game. The game executionenvironment 332 comprises active memory along with computing and videoprocessing. The game execution environment 332 receives gaming controls,such as controller input, through an I/O channel and causes the game tobe manipulated and progressed according to its programming. In oneembodiment, the game execution environment 332 outputs a rendered videostream that is communicated to the game device. Game progress may besaved online and associated with an individual person that has an IDthrough a gaming service. The game ID may be associated with a facialpattern.

The game data store 334 stores game code for various game titles. Thegame execution environment 332 may retrieve a game title and execute itto provide a gaming experience. Alternatively, the content distributioncomponent 338 may download a game title to an entertainment device, suchas entertainment device A 310.

The content data store 336 stores media titles, such as songs, videos,television shows, and other content. The distribution component 338 maycommunicate this content from content data store 336 to theentertainment devices 310-316. Once downloaded, an entertainment devicemay play the content on or output the content from the entertainmentdevice. Alternatively, the streaming component 340 may use content fromcontent data store 336 to stream the content to the person.

The content recognition database 342 includes a collection of audioclips associated with known media titles that may be compared to audioinput received at the entertainment service 330. As described above, thereceived audio input (e.g., received from the game console 210 of FIG.2) is mapped to the library of known media titles. Upon mapping theaudio input to a known media title, the source of the audio input (i.e.,the identity of media content) may be determined. The identified mediatitle/content is then communicated back to the entertainment device(e.g., the game console) for further processing. Exemplary processingmay include associating the identified media content with a person thatviewed or is actively viewing the media content and storing theassociation as a viewing record.

The entertainment service 330 also provides advertisements.Advertisements available for distribution may be stored within ad datastore 344. The advertisements may be presented as an overlay inconjunction with primary content and may be partial or full-screenadvertisements that are presented between segments of a mediapresentation or between the beginning and end of a media presentation,such as a television commercial. The advertisements may be associatedwith audio content. Additionally, the advertisements may take the formof secondary content that is displayed on a companion device inconjunction with a display of primary content. The advertisements mayalso be presented when a person associated with a targeted persona islocated in the audience area and/or is logged in to the entertainmentservice 330, as further described below.

The ad placement component 346 determines when an advertisement shouldbe displayed to a person and/or what advertisement should be displayed.The ad placement component 346 may consume real-time audience data andautomatically place an advertisement associated with a highest-biddingadvertiser in front of one or more viewers because the audience dataindicates that the advertiser's bidding criteria is satisfied. Forexample, an advertiser may wish to display an advertisement to menpresent in Kansas City, Mo. When the audience data indicates that one ormore men in Kansas City are viewing primary content, an ad could beserved with that primary content. The ad may be inserted into streamingcontent or downloaded to the various entertainment devices along withtriggering mechanisms or instructions on when the advertisement shouldbe displayed to the person. The triggering mechanisms may specifydesired audience data that triggers display of the ad.

The ad placement component 346 may manage linked advertisements. The adplacement component 346 may communicate preliminary advertisements andsubsequent advertisements to entertainment clients. For example, the adplacement component 346 could communicate a preliminary advertisementand associated response criteria to a smart TV. The smart TV couldindicate that an audience member satisfied the response criteria. Inresponse, the ad placement component 346 could communicate subsequentadvertisements to the audience member's tablet along with presentationtrigger. The ad placement component 346 could activate and deactivatesubsequent advertisements as view responses to the advertisements arereceived. Further, viewers' response to ads in an ad path may tracked bythe ad placement component 346. The ad placement component 346 maymaintain a record of viewer responses and purchases. The ad placementcomponent 346 may bill advertisers using the viewer response data.

The ad sales component 348 interacts with advertisers 320 to set a pricefor displaying an advertisement. In one embodiment, an auction isconducted for various advertising space. The auction may be a real-timeauction in which the highest bidder is selected when a viewer or viewingopportunity satisfies the advertiser's criteria.

The audience data store 350 aggregates and stores audience data receivedfrom entertainment devices 310-316. The audience data may first beparsed according to known types or titles of media content. Each item ofaudience data that relates to a known type or title of media content isa viewing record for that media content. Viewing records for each typeof media content may be aggregated, thereby generating viewing data. Theviewing data may be summarized according to categories. Exemplarycategories include a total number of persons that watched the content,the average number of persons per household that watched the content, anumber of times certain persons watched the content, a determinedresponse of people toward the content, a level of engagement of peoplein the media title, a length of time individuals watched the content,the common distractions that were ignored or engaged in while thecontent was being displayed, and the like. The viewing data maysimilarly be summarized according to types of persons that watched theknown media content. For example, personal characteristics of thepersons, demographic information about the persons, and the like may besummarized within the viewing data.

The audience processing component 352 may build and assign personasusing the audience data and a machine-learning algorithm. A persona isan abstraction of a person or groups of people that describespreferences or characteristics about the person or groups of people. Thepersonas may be based on media content the persons have viewed orlistened to, as well as other personal information stored in a userprofile on the entertainment device (e.g., game console) and associatedwith the person. For example, the persona could define a person as afemale between the ages of 20 and 35 having an interest in sciencefiction, movies, and sports. Similarly, a person that always has apositive emotional response to car commercials may be assigned a personaof “car enthusiast.” More than one persona may be assigned to anindividual or group of individuals. For example, a family of five mayhave a group persona of “animated film enthusiasts” and “footballenthusiasts.” Within the family, a child may be assigned a persona of“likes video games,” while the child's mother may be assigned a personof “dislikes video games.” It will be understood that the examplesprovided herein are merely exemplary. Any number or type of personas maybe assigned to a person.

The audience distribution component 354 may distribute audience data tocontent providers, advertisers, or other interested parties. Forexample, the audience distribution component 354 could provideinformation indicating that 300,000 discrete individuals viewed atelevision show in a geographic region. The audience data could bederived from image data received at each entertainment device. Inaddition to the number of people that viewed the media content, moregranular information could be provided. For example, the total personsgiving full attention to the content could be provided. In addition,response data for people could be provided. To protect the identity ofindividual persons, only a persona assigned to a person may be exposedand distributed to advertisers. A value may be placed on thedistribution, as a condition on its delivery, as described above. Thevalue may also be based on the amount, type, and dearth of viewing datadelivered to an advertiser or content publisher.

Turning now to FIG. 4, an audience area 400 that includes a group ofpeople is shown, in accordance with an embodiment of the presentinvention. The audience area is the area in front of the display device410. In one embodiment, the audience area 400 comprises the area fromwhich a person can see the content. In another embodiment, the audiencearea 400 comprises the area within a viewing range of the imaging device418. In most embodiments, however, the viewing range of the imagingdevice 418 overlaps with the area from which a person can see content onthe display device 410. If the content is only audio content, then theaudience area is the area where the person may hear the content.

Content is provided to the audience area by an entertainment system thatcomprises a display device 410, a game console 412, a cable box 414, aDVD player 416, and an imaging device 418. The game console 412 may besimilar to game console 210 of FIG. 2 described previously. The cablebox 414 and the DVD player 416 may stream content from an entertainmentservice, such as entertainment service 330 of FIG. 3, to the displaydevice 410 (e.g., television). The game console 412, cable box 414, andthe DVD player 416 are all coupled to the display device 410. Thesedevices may communicate content to the display device 410 via a wired orwireless connection, and the display device 410 may display the content.In some embodiments, the content shown on the display device 410 may beselected by one or more persons within the audience. For example, aperson in the audience may select content by inserting a DVD into theDVD player 416 or select content by clicking, tapping, gesturing, orpushing a button on a companion device (e.g., a tablet) or a remote incommunication with the display device 410. Content selected for viewingmay be tracked and stored on the game console 412.

The imaging device 418 is connected to the game console 412. The imagingdevice 418 may be similar to imaging device 213 of FIG. 2 describedpreviously. The imaging device 418 captures image data of the audiencearea 400. Other devices that include imaging technology, such as thetablet 212 of FIG. 2, may also capture image data and communicate theimage data to the game console 412 via a wireless or wired connection.

In one embodiment, audience data may be gathered through imageprocessing. Audience data may include a detected number of personswithin the audience area 400. Persons may be detected based on theirform, appendages, height, facial features, movement, speed of movement,associations with other persons, biometric indicators, and the like.Once detected, the persons may be counted and tracked so as to preventdouble counting. The number of persons within the audience area 400 alsomay be automatically updated as people leave and enter the audience area400.

Audience data may similarly include a direction each audience member isfacing. Determining the direction persons are facing may, in someembodiments, be based on whether certain facial or body features aremoving or detectable. For example, when certain features, such as aperson's cheeks, chin, mouth and hairline are detected, they mayindicate that a person is facing the display device 410. Audience datamay include a number of persons that are looking toward the displaydevice 410, periodically glancing at the display device 410, or notlooking at all toward the display device 410. In some embodiments, aperiod of time each person views specific media presentations may alsocomprise audience data.

As an example, audience data may indicate that an individual 420 isstanding in the background of the audience area 400 while looking at thedisplay device 410. Individuals 422, 424, 426, and child 428 and child430 may also be detected and determined to be all facing the displaydevice 410. A man 432 and a woman 434 may be detected and determined tobe looking away from the television. The dog 436 may also be detected,but characteristics (e.g., short stature, four legs, and long snout)about the dog 436 may not be stored as audience data because theyindicate that the dog 436 is not a person.

Additionally, audience data may include an identity of each personwithin the audience area 400. Facial recognition technologies may beutilized to identify a person within the audience area 400 or to createand store a new identity for a person. Additional characteristics of theperson (e.g., form, height, weight) may similarly be analyzed toidentify a person. In one embodiment, the person's determinedcharacteristics may be compared to characteristics of a person stored ina user profile on the display device 410. If the determinedcharacteristics match those in a stored user profile, the person may beidentified as a person associated with the user profile.

Audience data may include personal information associated with eachperson in the audience area. Exemplary personal characteristics includean estimated age, a race, a nationality, a gender, a height, a weight, adisability, a medical condition, a likely activity level of (e.g.,active or relatively inactive), a role within a family (e.g., father ordaughter), and the like. For example, based on the image data, an imageprocessor may determine that audience member 420 is a woman of averageweight. Similarly, analyzing the width, height, bone structure, and sizeof individual 432 may lead to a determination that the individual 432 isa male. Personal information may also be derived from stored userprofile information. Such personal information may include an address, aname, an age, a birth date, an income, one or more viewing preferences(e.g., movies, games, and reality television shows) of or logincredentials for each person. In this way, audience data may be generatedbased on both processed image data and stored personal profile data. Forexample, if individual 434 is identified and associated with a personalprofile of a 13-year-old, processed image data that classifiesindividual 434 as an adult (i.e., over 18 years old) may be disregardedas inaccurate.

The audience data also comprises an identification of the primarycontent being displayed when image data is captured at the imagingdevice 418. The primary content may, in one embodiment, be identifiedbecause it is fed through the game console 412. In other embodiments,and as described above, audio output associated with the display device410 may be received at a microphone associated with the game console412. The audio output is then compared to a library of known content anddetermined to correspond to a known media title or a known genre ofmedia title (e.g., sports, music, movies, and the like). As well, othercues (e.g., whether the person appears to be listening to as opposed towatching a media presentation) may be analyzed to determine the identityof the media content (e.g., a song as opposed to the soundtrack to amovie). Thus, audience data may indicate that basketball game 411 wasbeing displayed to individuals 420, 422, 424, 426, 428, 430, 432, and434 when images of the individuals were captured. The audience data mayalso include a mapping of the image data to the exact segment of themedia presentation (e.g., basketball game 411) being displayed when theimage data was captured.

Turning now to FIG. 5, an audience area depicting audience members'levels of engagement is shown, in accordance with an embodiment of thepresent invention. The entertainment system is identical to that shownin FIG. 4, but the audience members have changed. Image data captured atthe imaging device 418 may be processed similarly to how it wasprocessed with reference to FIG. 4. However, in this illustrativeembodiment, the image data may be processed to generate audience datathat indicates a level of engagement of and/or attention paid by theaudience toward the media presentation (e.g., the basketball game 411).

An indication of the level of engagement of a person may be generatedbased on detected traits of or actions taken by the person, such asfacial features, body positioning, and body movement. For example, themovement of a person's eyes, the direction the person's body is facing,the direction the person's face is turned, whether the person is engagedin another task (e.g., talking on the phone), whether the person istalking, the number of additional persons within the audience area 500,and the movement of the person (e.g., pacing, standing still, sitting,or lying down) are traits of and/or actions taken by a person that maybe distilled from the image data. The determined traits may then bemapped to predetermined categories or levels of engagement (e.g., a highlevel of engagement or a low level of engagement). Any number ofcategories or levels of engagement may be created, and the examplesprovided herein are merely exemplary.

In another embodiment, a level of engagement may additionally beassociated with one or more predetermined categories of distractions. Inthis way, traits of or actions taken by a person may be mapped to both alevel of engagement and a type of distraction. Exemplary actions thatindicate a distraction include engaging in conversation, using more thanone display device (e.g., the display device 510 and a companiondevice), reading a book, playing a board game, falling asleep, getting asnack, leaving the audience area 500, walking around, and the like.Exemplary distraction categories may include “interacted with otherpersons,” “interacted with an animal,” “interacted with other displaydevices,” “took a brief break,” and the like.

Other input that may be used to determine a person's level of engagementis audio data. Microphones associated with the game console 412 may pickup conversations or sounds from the audience. The audio data may beinterpreted and determined to be responsive to (i.e., related to ordirected at) the media presentation or nonresponsive to the mediapresentation. The audio data may be associated with a specific person(e.g., a person's voice). As well, signal data from companion devicesmay be collected to generate audience data. The signal data mayindicate, in greater detail than the image data, a type or identity of adistraction, as described below.

Thus, the image data gathered through imaging device 418 may be analyzedto determine that individual 520 is reading a paper 522 and is thereforedistracted from the content shown on display device 510. Individual 536is viewing tablet 538 while the content is being displayed throughdisplay device 510. In addition to observing the person holding thetablet, signal data may be analyzed to understand what the person isdoing on the tablet. For example, the person could be surfing the Web,checking e-mail, checking a social network site, or performing someother task. However, the individual 536 could also be viewing secondarycontent that is related to the primary content 411 shown on displaydevice 510. What the person doing on tablet 538 may cause a differentlevel of engagement to be associated with the person. For example, ifthe activity is totally unrelated (i.e., the activity is not secondarycontent), then the level of engagement mapped to the person's action(i.e., looking at the tablet) and associated with the person may bedetermined to be quite low. On the other hand, if the person is viewingsecondary content that compliments the primary content 511, then theindividual 536's action of looking at the tablet may be mapped to asomewhat higher level of engagement.

Individuals 532 and 534 are carrying on a conversation with each otherbut are not otherwise distracted because they are seated in front of thedisplay device 510. If, however, audio input from individuals 532 and534 indicate that they are speaking with each other while seated infront of the display device 510, their actions may be mapped to anintermediate level of engagement. Only individual 530 is viewing theprimary content 511 and not otherwise distracted. Accordingly, a highlevel of engagement may be associated with individual 530 and/or themedia content being displayed.

Determined distractions and levels of engagement of a person mayadditionally be associated with particular portions of image data, andthus, corresponding portions of media content. As mentioned elsewhere,such audience data may be stored locally on the game console 412 orcommunicated to a server for remote storage and distribution. Theaudience data may be stored as a viewing record for the media content.As well, the audience data may be stored in a user profile associatedwith the person for whom a level of engagement or distractions wasdetermined.

Turning now to FIG. 6, a person's reaction to media content isclassified and stored in association with the viewing data. Theentertainment setup shown in FIG. 6 is the same as that shown in FIG. 4.However, the primary content 611 is different. In this case, the primarycontent is a car commercial indicating a sale. In addition to detectingthat individuals 620 and 622 are viewing the content and are paying fullattention to the content, the persons' responses to the car commercialmay be measured through one or more methods and stored as audience data.

In one embodiment, a person's response may be gleaned from the imagesand/or audio originating from the person (e.g., the person's voice).Exemplary responses include smiling, frowning, wide eyes, glaring,yelling, speaking softly, laughing, crying, and the like. Otherresponses may include a change to a biometric reading, such as anincreased or a decreased heart rate, facial flushing, or pupil dilation.Still other responses may include movement, or a lack thereof, forexample, pacing, tapping, standing, sitting, darting one's eyes, fixingone's eyes, and the like. Each response may be mapped to one or morepredetermined emotions, such as happiness, sadness, excitement, boredom,depression, calmness, fear, anger, confusion, disgust, and the like. Forexample, when a person frowns, her frown may be mapped to an emotion ofdissatisfaction or displeasure. In embodiments, mapping a person'sresponse to an emotion may additionally be based on the length of timethe person held the response or the pronouncement of the person'sresponse. As well, a person's response may be mapped to more than oneemotion. For example, a person's response (e.g., smiling and jumping upand down) may indicate that the person is both happy and excited.Additionally, the predetermined categories of emotions may include tiersor spectrums of emotions. Baseline emotions of a person may also betaken into account when mapping a person's response to an emotion. Forexample, if the person rarely shows detectable emotions, a detected“happy” emotion for the person may be elevated to a higher “tier” ofhappiness, such as “elation.” As well, the baseline may serve to informdeterminations about the attentiveness of the person toward a particularmedia title.

In some embodiments, only responses and determined emotions that areresponsive to the media content being displayed to the person areassociated with the media content. Responsiveness may be related to adetermined level of engagement of a person, as described above. Thus,responsiveness may be determined based on the direction the person islooking when a title is being displayed. For example, a person that isturned away from the display device is unlikely to be reacting tocontent being displayed on the display device. Responsiveness maysimilarly be determined based on the number and type of distractionslocated within the viewing area of the display device. Similarly,responsiveness may be based on an extent to which a person isinteracting with or responding to distractions. For example, a personwho is talking on the phone, even though facing and looking at a displayscreen of the display device, may be experiencing an emotion unrelatedto the media content being displayed on the screen. As well,responsiveness may be determined based on whether a person is activelyor has recently changed a media title that is being displayed (i.e., aperson is more likely to be viewing content he or she just selected toview). It will be understood that responsiveness can be determined inany number of ways by utilizing machine-learning algorithms, and theexamples provided herein are meant only to be illustrative.

Thus, returning to FIG. 6, the image data may be utilized to determineresponses of individual 622 and individual 620 to the primary content611. Individual 622 may be determined to have multiple responses to thecar commercial, each of which may be mapped to the same or multipleemotions. For example, the individual 622 may be determined to besmiling, laughing, to be blinking normally, to be sitting, and the like.All of these reactions, alone and/or in combination, may lead to adetermination that the individual 622 is pleased and happy. This isassumed to be a reaction to the primary content 611 and recorded inassociation with the display event. By contrast, individual 620 is notsmiling, has lowered eyebrows, and is crossing his arms, indicating thatthe individual 620 may be angry or not pleased with the car commercial.

Turning now to FIG. 7, a linear ad path 700 is shown, in accordance withan embodiment of the present invention. The linear ad path 700 includesa preliminary ad 710, a subsequent ad 720 having incentive A, and asubsequent ad 730 having incentive B. Incentives A and B are different.

The preliminary ad 710 may be shown as part of a media presentation,such as product placement in a television show or an ad shown during acommercial break. The preliminary ad 710 is associated with one or morereaction criteria that are used to determine whether an audience membershould be shown either of the subsequent ads. For example, thepreliminary advertisement 710 may require that the user pays fullattention to the preliminary ad to activate either subsequent ad. Usingattentiveness as the reaction criteria may be used when the ads build oneach other to tell a story, require knowledge of the previous ad in thepath, or understand the new ad.

In another embodiment, the reaction criteria specifies that a positiveresponse is detected or received from the audience member. An explicitresponse may be received while an implicit response is detected. Oncethe reaction criteria are satisfied, the subsequent ad 720 is activatedfor presentation to the user upon satisfaction of presentation triggersassociated with the subsequent ad 720.

The subsequent ad 720 may be communicated to a device associated withthe audience member. For example, the audience member may be associatedthrough a user account with a personal computer, tablet, and smartphone.The subsequent ad 720 may be communicated to one or more devices thatare capable of detecting context associated with the presentationtrigger, including the device on which the preliminary ad 710 wasviewed. For example, if the presentation trigger requires the user to bein a geographic area, then the subsequent ad 720 would only becommunicated to devices that are location aware. On the other hand, ifthe presentation trigger associated with subsequent ad 720 only requiresthat it be shown to the user at a particular time, then it could also besent to the personal computer, game console, or other nonlocation-awareentertainment devices.

The subsequent ad 720 may be shown to the user multiple times acrossmultiple devices. In each case, the presentation and response, if any,may be communicated to a centralized ad tracking service. At some point,the user's response, or lack of response, to the subsequent ad 720 maycause the user to be shifted down the ad path 700 to subsequent ad 730having incentive B. In one embodiment, the failure of the user torespond to subsequent ad 720 with incentive A causes the user to beshifted down the path 700 to subsequent ad 730 with incentive B, whichis higher than incentive A. In other situations, incentive A andincentive B are not of a significantly different value, but are justdifferent. For example, incentive A could be for the user to get a freesoft drink, while incentive B is for the user to get a free cup ofcoffee.

In one embodiment, the user's positive response to subsequent ad 720causes the user to be shifted to subsequent ad 730. For example, upondetecting that the user purchased a first product in response tosubsequent ad 720, a related product could be advertised throughsubsequent ad 730. In this case, incentives A and B would be directedtoward different products associated with their correspondingadvertisements. For example, having purchased movie tickets throughsubsequent ad 720, subsequent ad 730, having a coupon for popcorn, couldbe displayed upon the user arriving at the theater.

Turning now to FIG. 8, a nonlinear ad path 800 is shown, in accordancewith an embodiment of a present invention. The ad path 800 starts with apreliminary user response 810 to content, such as preliminary ad.Different subsequent ads within the ad path 800 may be activated inresponse to the preliminary user response 810. The preliminary userresponse 810 may be explicit or implicit. The preliminary user response810 may be in response to an advertisement, but could also be inresponse to nonadvertising content.

For example, the user could respond positively to a baseball game(nonadvertising content) between two teams. Upon determining that theuser is associated with a city where one of the teams is based, the usercould be placed into an ad path designed to incentivize the user topurchase goods or services associated with the baseball team. Forexample, subsequent ad A 812, having presentation trigger A could berelated to the user purchasing tickets for a baseball game. At decisionpoint 814, the user's response to subsequent ad A 812 is evaluated. Upondetermining that the user purchased baseball tickets, the path may bedeactivated at step 816. The user's purchase record may be updatedindicating that the user purchased baseball tickets.

Upon determining that the user did not purchase baseball tickets inresponse to subsequent ad A 812, the user may be moved to a differentpart in the path 800 associated with subsequent ad B 818. As can beseen, ad 818 is associated with a different trigger and could also beassociated with different incentives. The response to subsequent ad B818 is monitored at decision point 820. Upon determining that a positiveresponse has not been received, the ad B 818 may remain active forsubsequent presentation when trigger B is satisfied the next time. If apositive response is noted at decision point 820, the user could bemoved to a different part in the path associated with subsequent ad E822. Notice that subsequent ad B 818 and subsequent ad E 822 are bothassociated with trigger B. Ads within an advertising path and acrossdifferent advertising paths may use the same triggers. For example,trigger B could be associated with a time frame before an upcomingbaseball home stand. The subsequent ad E 822 could be associated with adifferent home stand or games that received the positive response tosubsequent ad B 818. Though not shown, the various points along the pathcould loop or be deactivated in response to a positive response orpurchase.

In one embodiment, the part of the path showing subsequent ad C 824 andsubsequent ad F 826 are related to a complimentary product, such as abaseball jersey or cap. Thus, while the part of the path associated withsubsequent ads 812, 818, and 822 are all attempting to sell baseballtickets, the complimentary products may be part of a related subsequentpath with different triggers and incentives. For example, the trigger Cthat is part of subsequent ad C 824 may be related to geographicproximity with a retail outlet where baseball caps are sold.

The user could be associated with multiple subsequent ads within the adpath 800 at the same time when appropriate. For example, the user couldbe associated with a subsequent ad offering baseball tickets at the sametime she is associated with a subsequent ad selling baseball caps.Similarly, the user could be associated with multiple subsequent adsoffering the same thing but with different triggers. For example, thetriggers could specify different geographic locations associated withdifferent retail stores and different incentives offered by thoserespective stores.

Turning now to FIG. 9, a method 900 of providing linked advertisementsis shown, in accordance with an embodiment of the present invention. Themethod may be performed on a game console or other entertainment devicethat is connected to an imaging device with a view of an audience areaapproximate to a display device.

At Step 910, image data that depicts an audience for an ongoing mediapresentation is received. The image data may be in the form of a depthcloud generated by a depth camera, a video stream, still images,skeletal tracking information or other information derived from theimage data. The ongoing media presentation may be a movie, game,television show, an advertisement, or the like. Ads shown during breaksin a television show may be considered part of the ongoing mediapresentation.

The audience may include one or more individuals within an audiencearea. The audience area includes the extents from which the ongoingmedia presentation may be viewed from the display device. Theindividuals within the audience area may be described as audiencemembers herein.

At step 920, an audience member is identified by analyzing the imagedata. In one embodiment, the audience member is identified throughfacial recognition. For example, the audience member may be associatedwith a user account that provides facial recognition authentication orlogin. The audience member's account may then be associated with one ormore social networks. In one embodiment, social networks are associatedwith a facial recognition login feature that allows the audience memberto be associated with a social network.

The audience member may be given an opportunity to explicitly associatehis account with one or more social networks. The audience member may bea member of more social networks than are actually associated with theaccount. But embodiments of the present invention may work with whateversocial networks the audience member has provided access to. Upondetermining that the audience member is associated with a socialnetwork, the audience member may be asked to provide authenticationinformation or permission to access the social network. This informationmay be requested through a setup overlay or screen. The setup may occurat a point separate from when the media presentation is ongoing, forexample, when an entertainment device is set up.

At Step 930, audience data is generated by analyzing the image data.Exemplary audience data has been described previously. The audience datamay include a number of people that are present within the audience. Forexample, the audience data could indicate that five people are presentwithin the audience area. The audience data may also associate audiencemembers with demographic characteristics.

The audience data may also indicate an audience member's level ofattentiveness to the ongoing media presentation. Different audiencemembers may be associated with a different level of attentiveness. Inone embodiment, the attentiveness is measured using distractionsdetected within the image data. In other words, a member's interactionswith objects other than the display may be interpreted as the memberpaying less than full attention to the ongoing media presentation. Forexample, if the audience member is interacting with a different mediapresentation (e.g., reading a book, playing a game) then less than fullattentiveness is paid to the ongoing media presentation. Interactionswith other audience members may indicate a low level of attentiveness.Two audience members having a conversation may be assigned less than afull attentiveness level. Similarly, an individual speaking on a phonemay be assigned less than full attention.

In addition to measuring distractions, an individual's actions inrelation to the ongoing media presentation may be analyzed to determinea level of attentiveness. For example, the user's gaze may be analyzedto determine whether the audience member is looking at the display. Whenmultiple content items are shown within the ongoing media presentation,such as an overlay over a primary content, gaze detection may be used todetermine whether the user is ignoring the overlay and looking at theongoing media presentation or is focused on the overlay, or even noticedthe overlay for a short period. Thus, attentiveness information could beassigned to different content shown on a single display.

The audience data may also measure a user's reaction or response to theongoing media presentation. As mentioned previously with reference toFIG. 6, a user's response or reaction may be measured based on biometricdata and facial expressions.

At step 940, the audience member is determined to have reactedpositively to a preliminary advertisement for a product or service shownas part of the ongoing media presentation. The preliminary advertisementcould be a commercial shown during a break in the primary content,including before presentation of the primary content begins or after itconcludes. The preliminary advertisement could also be product placementwithin primary media content. The preliminary advertisement could be anoverlay. The overlay could be shown concurrently with the primarycontent.

The audience member's positive reaction is determined using the audiencedata generated previously at step 930. In one embodiment, reactionswithin the audience data are correlated to content within the ongoingmedia presentation. For example, a positive response observed at thesame time a sports car appears within the media content may trigger asubsequent ad for the sports car. The sports car may be part of aproduct placement.

In one embodiment, each reaction within the audience data is associatedwith a time and can be used to associate the reaction with a particularpoint in the content. For example, if the presentation starts at noonand a reaction is observed within the audience data at 1:00 p.m., thenthe reaction may be associated with content shown one hour into theongoing media presentation. Other ways to correlate a reaction with apoint within the media presentation are possible. In one embodiment,metadata is associated with the ongoing media presentation to identifycontent displayed at various progress points. One example of content isa preliminary advertisement. A preliminary advertisement may be atelevision commercial, product placement, an overlay, or the like.

The preliminary advertisement is associated with a response or reactionthreshold. When a reaction within the audience data is correlated to thepreliminary advertisement and determined to satisfy the reactionthreshold, then a subsequent advertisement may be activated. Asubsequent advertisement may be thought of as a follow-up to thepreliminary advertisement. In one embodiment, the preliminaryadvertisement and subsequent advertisement could be the same. However,the subsequent advertisement is associated with one or more presentationtriggers. The subsequent ad is only displayed when the presentationtrigger criteria are satisfied. The presentation criteria may be contextbased. For example, the subsequent advertisement may be shown on amobile device associated with the audience member at a time and placethat is conducive to purchasing an advertised product or service.

At step 950, the audience member is associated with an ad path thatcomprises at least one subsequent advertisement that has a presentationtrigger that when satisfied causes a presentation of the subsequentadvertisement. Examples of ad paths have been described previously. Theuser could be associated with a particular subsequent advertisementwithin the ad path that includes multiple subsequent advertisements. Inone embodiment, the strength of the user reaction is determined or isused to determine which subsequent advertisement within the path shouldbe activated. An activated subsequent advertisement is activelymonitored for satisfaction of the presentation trigger associated withthe subsequent advertisement. Dormant or inactive subsequentadvertisements within the path are not monitored and are not triggeredfor display until activated. A user's response to a first subsequentadvertisement, either positive or negative, could cause an additionalsubsequent advertisement to be activated and the initial subsequentactive advertisement to be inactivated.

In one embodiment, the audience member is associated with a useraccount. The user account may be for an entertainment service, e-mail, asocial network, or other service that is accessed through multipledevices. The subsequent advertisements may be presented through thisservice or through an application associated with this service. In thisway, subsequent advertisements may be presented on one or more devicesassociated with the user. Accordingly, the ad path may be communicatedto multiple devices associated with the user. Different subsequentadvertisements could be activated on different devices. For example, asubsequent advertisement associated with a particular time and locationcould be active on the mobile device associated with the audiencemember, whereas a different subsequent advertisement could be activatedon a game console or other mostly stationary device. In one embodiment,the subsequent advertisement on the mobile device is obtrusive. Thesubsequent advertisement may be associated with a vibration, noise, orother indication to get the user's attention.

Turning now to FIG. 10, a method 1000 of assigning an audience member toan ad path is shown, in accordance with an embodiment of the presentinvention. At step 1010, image data that depicts an audience for theongoing media presentation is received. At step 1020, an audience memberis identified by analyzing the image data. For example, the audiencemember could be identified using voice recognition or facialrecognition. At step 1030, audience data is generated by analyzing theimage data. The audience data may classify the reactions of individualspresent within the audience. The individuals may be alternativelydescribed as users or audience members.

At step 1040, a strength of the audience member's reaction to contentshown as part of the ongoing media presentation is determined from theaudience data. The strength of the reaction is intended to capture auser's enthusiasm for a content, such as a preliminary advertisement.The strength of the advertisement may be determined based on an audiencemember profile that tracks a range of responses made by the audiencemember. For example, a raised eyebrow may be an extremely positivestrong response from a first audience member and only a mildly positiveresponse or even a skeptical response from a different audience member.Some individuals are more expressive than others and the user accountand profile is able to adjust for these differences by comparing thereactions of an individual over time. Feedback from an advertising pathmay be provided to further refine the responses of an individual. Forexample, if a particular reaction is initially interpreted as positive,but the user never responds positively to the advertisements, then theexpression may be reclassified.

At step 1050, the audience member is associated with an ad path thatcomprises multiple subsequent advertisements that each has apresentation trigger that, when satisfied, causes a presentation of anassociated subsequent advertisement. An ad path having multiplesubsequent advertisements has been described previously with referenceto FIG. 8. In one embodiment, the ad path, including the subsequentadvertisements and associated presentation triggers, are communicated tomultiple devices associated with the audience member. For example, thecommunication or the ad path could be communicated to the audiencemember's smartphone and tablet. The communications between devices couldbe coordinated through a central advertising component associated withan advertising service or entertainment service. The centralized adservice may generate different ad paths and communicate them todifferent devices. The centralized ad service may also download initialtriggers to entertainment devices that cause the user to be associatedwith a certain ad path.

In another embodiment, audience data is communicated from anentertainment device to a centralized ad component that analyzes theaudience data and associates the user with an ad path when variouscriteria are satisfied. In an alternative embodiment, the criteria arecommunicated to an entertainment device that makes the comparison.

Turning now to FIG. 11, a method 1100 of managing an ad path is shown,in accordance with an embodiment of the present invention. Method 1100may be performed by a mobile device such as a smartphone or a tablet. Inone embodiment, the mobile device is location-aware through GPStechnology, or other location technology.

At step 1110, an ad path that comprises a subsequent advertisement thatis related to content to which the user previously responded positivelyis received on a mobile device associated with a user. The subsequentadvertisement could have been shown on a different device such as atelevision or on the mobile device.

At step 1120, determining that a presentation trigger associated withthe subsequent advertisement is determined to be satisfied by the mobiledevice's present context. The context could include a time, location,and user activity. An example of a user activity includes driving,riding a bus, and riding a train. In each case, the user context fortime and location could be satisfied, but the user may only be shown theadvertisement if driving. This may make sense because the user may notbe able to get off a train or a bus in time to respond to the suggestionmade within the subsequent advertisement. For example, the subsequentadvertisement could provide a coupon for coffee as the user approaches acoffee shop. If the user was determined to be on a bus, for example, byobserving a pattern of starting and stopping at known bus stops, thenthe user may not be shown the advertisement unless a bus stop is nearthe coffee shop. All of this would be taken into account by thepresentation triggers.

At step 1130, the subsequent advertisement is presented to the user. Inone embodiment, the user's response to the subsequent advertisement ismonitored. A positive or negative response may be communicated to acentral ad component that handles billing and may provide additionalinstructions regarding the next steps with regard to the ad path. In oneembodiment, a different subsequent advertisement within the ad path isactivated upon detecting a positive or a negative response. The presentsubsequent advertisement may be simultaneously deactivated.

Embodiments of the invention have been described to be illustrativerather than restrictive. It will be understood that certain features andsubcombinations are of utility and may be employed without reference toother features and subcombinations. This is contemplated by and iswithin the scope of the claims.

The invention claimed is:
 1. One or more computer-storage device havingcomputer-executable instructions embodied thereon that when executed bya computing device perform a method of providing linked advertisements,the method comprising: receiving image data that depicts an audience foran ongoing media presentation; identifying an audience member byanalyzing the image data; generating audience data by analyzing theimage data; determining from the audience data that the audience memberhas reacted positively to a preliminary advertisement for a product orservice shown as part of the ongoing media presentation; in response tosaid determining, associating the audience member with an ad path thatcomprises a subsequent advertisement that has a presentation triggerthat, when satisfied, causes a presentation of the subsequentadvertisement, monitoring the user's response to the subsequentadvertisement, activating a different subsequent advertisement in the adpath based on the user's response, and monitoring conditions todetermine when the presentation trigger associated with the differentsubsequent advertisement is satisfied, and wherein the user's responseis negative and the different subsequent advertisement comprises alarger incentive for the user to purchase an advertised good or service.2. The device of claim 1, wherein the method further comprisesdetermining an audience member's reaction to the preliminaryadvertisement is positive by analyzing the image data and including theaudience member's reaction in the audience data.
 3. The device of claim1, wherein the method further comprises associating the audience memberwith a user account and associating the audience member with a mobiledevice through the user account.
 4. The device of claim 3, wherein themethod further comprises determining that the presentation trigger hasbeen satisfied using data generated by the mobile device and presentingthe subsequent advertisement to the audience member.
 5. The device ofclaim 4, wherein the method further comprises presenting the subsequentadvertisement through the mobile device.
 6. The device of claim 1,wherein the presentation trigger is satisfied when the audience memberis near a retail outlet where the product or service is sold.
 7. Thedevice of claim 1, wherein determining that the audience member hasreacted positively to the preliminary advertisement comprisesrecognizing an affirmative gesture performed by the audience member. 8.The device of claim 1, wherein the method further comprises determiningdemographic information for the audience member by analyzing the imagedata and including the demographic information within the audience data.9. A method of assigning an audience member to an ad path, the methodcomprising: receiving image data that depicts an audience for an ongoingmedia presentation; identifying the audience member by analyzing theimage data; generating audience data by analyzing the image data;determining, from the audience data, a strength of the audience member'sreaction to content shown as part of the ongoing media presentation; inresponse to said determining, associating the audience member with an adpath that comprises multiple subsequent advertisements that each has apresentation trigger that, when satisfied, causes a presentation of anassociated subsequent advertisement, wherein the ad path comprises ahigh-enthusiasm subsequent advertisement that is associated with alower-level incentive and a low-enthusiasm subsequent advertisement thatis associated with a higher-level incentive; and assigning the audiencemember to the high-enthusiasm subsequent advertisement when the strengthof the audience member's reaction is strongly positive and thelow-enthusiasm subsequent advertisement when the strength of theaudience member's reaction is weakly positive.
 10. The method of claim9, wherein the audience member is identified through facial recognitionusing the image data.
 11. The method of claim 9, wherein the strength ofthe audience member's reaction is determined using a reaction profilefor the audience member that includes a range or reactions observed inthe audience member.
 12. The method of claim 9, wherein the content isnot an advertisement.
 13. One or more computer-storage device havingcomputer-executable instructions embodied thereon that when executed bya computing device perform a method of managing an ad path, the methodcomprising: receiving, on a mobile device associated with a user, an adpath that comprises a subsequent advertisement that is related tocontent to which the user previously responded positively; determiningthat a presentation trigger associated with the subsequent advertisementis satisfied by the mobile device's present context; presenting thesubsequent advertisement to the user, monitoring the user's response tothe subsequent advertisement, activating a different subsequentadvertisement in the ad path based on the user's response, andmonitoring conditions to determine when the presentation triggerassociated with the different subsequent advertisement is satisfied, andwherein the user's response is negative and the different subsequentadvertisement comprises a larger incentive for the user to purchase anadvertised good or service.
 14. The device of claim 13, wherein the userresponded positively by performing a gesture.
 15. The device of claim13, wherein the ad path comprises multiple subsequent ads that areindividually activated based on different user responses to one or moreof the multiple subsequent ads.
 16. The device of claim 13, wherein thepresentation trigger comprises the mobile device being located in ageographic area.